Online News

Business Model of online News

• Does the business model have an effect on the editorial content?
• Methodology:

1. Estimate a Structural Topic Model
2. Use posterior distribution to estimate the effect of document metadata.

Data

Online news articles about domestic politics from 01.06.2017 - 01.12.2017

Concepts

• A single observation in a textual database is called a document.

• The set of documents that make up the dataset is called a corpus.

• Covariates associated with each document are called metadata.

Data Structure

• Documents (articles) are stored on a “Document-Term-Matrix”

• Documents are seen as “bag of words”

• Each article has metadata: publisher (news platform) and the day it was published.

How to find out latent topics in an article?

Topic Model

Credits: Christine Doig

The intuition behind LDA

Credits: Blei (2012)

LDA as a graphical model

• Nodes are random variables; arrows indicate dependence
• Plates indicate replicated variables:
  • \(N =\) collection of words within a document.
  • \(D =\) collection of documents within a corpus.
• Shaded nodes are observed; unshaded nodes are hidden
  • observed: word in a document \(w_{d,n}\)
  • fixed: mixture components (number of topics \(K\) & vocabulary)
  • hidden: mixture proportions (per-document topic proportions \(\theta_d\) & word-topic distribution \(\beta_k\))

LDA - joint distribution

\[ \begin{aligned} p(\beta_{1:K},\theta_{1:D},z_{1:D}, w_{1:D}) \propto \end{aligned} \]

\[ \begin{aligned} \displaystyle \prod_{i=1}^{K}p(\beta_i)\displaystyle \prod_{d=1}^{D}p(\theta_d)(\prod_{n=1}^Np(z_{d,n,}|\theta_d)p(w_{d,n}|\beta_{1:K},z_{d,n})) \end{aligned} \]

Generative process

• \(K\): choose the number of topics

  • \(K=3\)

• \(\theta_d\): for each document \(d\), choose a distribution over topics;

  • \(\theta_d\) ~ Dirichlet(\(\alpha\))

• \(z_{d,n}|\theta_d\): according to \(\theta_d\), assign a topic \(z_{d,n}\) for the \(n^{th}\) word;

  • \(K=Topic 1\)

Generative process (contd.)

• \(w_{d,n}|z_{d,n},\beta,\theta\): choose a term from that topic according to \(\beta_k\)

  • \(\beta_k\) ~ Dirichlet(\(\eta\))

• \(N\): repead this process for all \(n\) word-positions in the document.

• \(D\): conduct this process for all \(d\) documents in the corpus

Research Process

Strucutral Topic Model (Roberts et. al. (2016))

• Including covariates into a topic model:

1. Topic Prevalence: Attributes that affect the likelihood of discussing topic \(k\)

   News platform & the month the article was published.

2. Topic Content: Attributes that affect the likelihood of including term \(w\) overall, and of including it within topic \(k\)

   Not yet included.

STM Priors

Bayesian inference

Posterior probability:

\[ \begin{aligned} p(\eta,z,\kappa,\gamma,\Sigma|w,X,Y) \propto \displaystyle \prod_{d=1}^{D}(\eta_d|X_d\gamma,\Sigma) (\displaystyle \prod_{n=1}^{D}(z_{n,d}|\theta_d)*(w_n|\beta_{d,k=z_{d,n}}))) * \displaystyle \prod p(\kappa)\displaystyle \prod p(\Gamma) \end{aligned} \]

• The number of possible topic structures is exponentially large –> sum is intractable to compute.
• Instead of obtaining a closed-form solution for the posterior distribution, we must approximate it.

Approximate Posterior

Central research goal of probabilistic modeling: develop efficient methods for approximating posterior.

• Mean field variational methods (Blei et al., 2001, 2003)
• Expectation propagation (Minka and Lafferty, 2002)
• Collapsed Gibbs sampling (Griffiths and Steyvers, 2002)
• Distributed samplung (Newsman et al., 2008; Ahmed et al., 2012)
• Stochastic inference (Hoffman et al., 2010, 2013; Mimno et al., 2012)
• Factorization inference (Arora et al., 2012; Anandkumar et al., 2012)
• Variational EM algorithm (Wang and Blei 2013; Roberts et. al. (2016)
• …

Model Selection

Prior Specification:

\(\gamma_{p,k}\) ~ Normal\((0,\sigma^2_k)\)

\(\sigma^2_k\) ~ Inverse-Gamma\((a,b)\)

Topic Selection:

\(K=28\)

Topic Proportions

Timeline

Sample Articles

Sample Articles (contd)

Estimate Effect of Covariates

Estimate the conditional expectation of topic prevalence for given document characteristics (lm on compositional data):

\[ \theta_d=\alpha+\beta_1 x_{site}+\beta_2x_{month}+\epsilon \]

Whats next?

1. Estimate model, including the effect of covariates on topical content How are topics discused within different newswires?

2. Relationship between topics How topics are correlated differently for different newswires, indicating how topics are connected and framed differently in each newswire.